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SYSTEMS AND METHODS FOR COMPUTING THE PRESENCE OF 
SELF-LUMINOUS ELEMENTS IN AN IMAGE 

TECHNICAL FIELD 

[01] The present invention relates generally to digital image processing and more particularly 
to the detection of aperture colors or self-luminous objects in an image. 

BACKGROUND 

[02] The reflected light reaching an image sensor is dependent on the light that illuminates the 
subject matter of the image. Different illuminants will produce different sensor values from the 
surfaces of the image subject matter. The human visual system approximately compensates these 
shifts in reflected light to a perceived norm such that surface colors appear color constant. 
However, when images are captured on media and viewed under a light source different than the 
source in the image scene, these natural corrections do not take place. Accordingly, it is often 
desirable for recorded images to be color-balanced to a reference light source in order to appear 
as they would to the natural eye. This balancing or color correction can be performed once the 
scene illuminant is identified. 

[03] There are many known methods for identification of a light source or illuminant in an 
image scene. However, the conventional correction algorithms assume that all image pixels 
represent reflecting surfaces. When an image contains self-luminous objects such as sky and 
other light sources, the surface-pixel assumption is violated. When an image contains a 
significant portion of non-reflective, self-luminous objects, conventional methods will fail and 
the image illuminant will be incorrectly determined. For example, if an image contains blue sky 
and the color-balance algorithm assumes that aU pixels are reflective objects, "bluish" pixels 
could be taken as evidence that the illumination of the scene is bluish. Because a color 
correction is approximately the opposite hue of the estimated illuminant, the correction for a 
bluish illuminant would be to shift the image in a yellowish direction. This correction might 
produce an overly yellowish ground/surface region and a desaturated sky region. 
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[04] These color correction, color balance or color constancy algorithms generally do not 
address the question of how to handle images containing luminous objects, which are also 
referred to herein as self-luminous objects. They have, rather, focused on images where the 
surface-pixel assumption is satisfied (e.g., uniformly illuminated Mondrian-like images). 
[05] It would be advantageous if a method existed that permitted accurate digital image 
illuminant corrections to be made for images containing significant exception regions, such as 
self-luminous regions. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

[06] The present embodiments will become more fiilly apparent from the following 
description and appended claims, taken in conjunction with the accompanying drawings. 
Understanding that these drawings depict only typical embodiments and are, therefore, not to be 
considered limiting of the invention's scope, the embodiments will be described with additional 
specificity and detail through use of the accompanying drawings in which: 

Fig. la is a flowchart illustrating methods of self-luminous element detection of some 
embodiments of the present invention; 

Fig. lb is a flowchart illustrating methods of self-luminous element detection and element 
weighting of some embodiments of the present invention; 

Fig. 2 is a an exemplary image used to illustrate elements of embodiments of the present 
invention; 

Fig, 3a is an image illustrating a likelihood of being self-luminous based on pixel position 
relative to an image boundary; 

Fig, 3b is an image illustrating a likelihood of being self-luminous based on color 
characteristics; 

Fig, 3c is an image illustrating a likelihood of being self-luminous based on luminance 
characteristics; 

Fig, 3d is an image illustrating a combined likelihood of being self-luminous based on the 
combination of factors illustrated in figures 3a, 3b and 3c; 

Fig. 4 is a diagram illustrating an exemplary self-luminous probability distribution 

Fig. 5 is a flowchart illustrating some steps of embodiments of the present invention; 

Fig. 6 is a diagram illustrating the various color distributions reflected from selected 
illuminants; 

Fig. 7 is a diagram illustrating the chromaticities of the illuminants shown in Fig. 6; 
Fig. 8 is a flowchart illustrating methods of embodiments employing a weighting factor 
related to a likelihood of being self-luminous; 

Fig. 9 is a flowchart illustrating methods of embodiments employing a correction factor; 

and 

Fig. 10 is a flowchart iUustrating methods of embodiments wherein self-luminous 
elements are distinguished during correction. 
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DETAILED DESCRIPTION 

[07] In some embodiments of the present invention, methods for determining image self- 
luminous regions are employed. Digital image correction parameters may be calculated in 
response to determining the self-luminous regions. Image self-luminous regions can be 
determined from image feature measures. For example, image feature measures can be pixel 
color characteristics, pixel luminance and/or pixel position parameters. In some embodiments, 
self-luminous regions may be determined in response to bright pixel luminance, where bright 
pixel luminance is defined as relatively bright with respect to surrounding image regions and/or 
brighter than a threshold brightness. 

[08] In some embodiments, self-luminous regions may be determined using a general function 
of color characteristics with a plurality of terms such as a function of chromaticity, hue angle, 
color saturation, or brightness. 

[09] In other embodiments, self-luminous regions may be determined in relation to the 
position of pixels with respect to the image edges or other geometric image characteristics. 
[10] In many algorithms of embodiments of die present invention, the correction takes the 
form of a matrix that is applied in a linear colorspace such as the tristimulus values of CIE 1931 
XYZ. Input tristimulus values can be multiplied by a matrix to produce a set of color-corrected 
values as follows: 

where 

X is a vector of tristimulus values 

M is a color correction matrix 

x' is the vector of corrected values. 
The color correction can be performed in other color spaces, including linear transformations of 
CIE 1931 XYZ or RGB sensor spaces. 
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[11] One view of color correction is that the critical step is estimating an illuminant for the 
scene and then from this estimate a correction matrix can be defined. This approach seeks to 
estimate surface reflectance functions and the illumination at each pixel The original sensor 
values can then be mapped to values under a canonical illuminant. This process assumes that the 
surface-pixel assumption is satisfied. 

[12] One method of defining a correction matrix is in terms of linear models for reflectances 
and illuminants and an estimated illuminant for the scene.. After performing a principle 
components analysis (or singular value decomposition, SVD) on a large set of reflectance or 
illuminant functions (e.g., those of the matte Munsell chipset or measured daylight power 
spectra), typical illuminants and surfaces can be approximated using low-dimensional linear 
models. Functions can then be approximated within the linear model by defining sets of weights: 

« = BeWe 

where 

r is a 1-by-nWavelengths vector of approximated reflectance values; 

e is a 1-by-nWavelengths vector of approximated illuminant spectral power values; 

Ws is a set of weights within the surface linear model; 

We is a set of weights within the illuminant linear model; 

Bs is a set of linear basis functions for surface reflectance; and, 

Be is a set of linear basis functions for illuminant spectral power. 

[13] Colorspace coordinates, x, can be computed for a reflectance function rendered under an 
illuminant. 

x = TEB,w,, 

where 

E is a diagonal [nWavelengths, nWavelengths] matrix representing the illuminant, 
i.e., E = diag(B^wJ. 

T is a matrix representing the conversion from wavelength domain to color 
coordinates (e.g., CIE 1931 color matching functions or a set of sensor response 
functions). 
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This equation for computing sensor values describes the physics of reflecting an illuminant 
spectral power distribution from a matte reflecting surface and applying the signal to a set of 
sensors. The dimensions of the bases are [nWavelengths, nBs] and [nWavelengths, nBe]. 
[14] Given an illuminant estimate for a first scene, it is possible to invert the mapping from 
color coordinates to surface weights. The surface weights may then be re-rendered under an 
illuminant in a second scene (e.g., a standard illumination condition such as CIE D6500). The 
color-corrected values under the second illuminant, x', can be computed as: 

x=Mx 

where 

M = TE2Bs(TEiBsr* 
E2 = diag(e^3^^^^^) 

Ej = diag(BEWE) 

^canonical spectral power distribution of a canonical illuminant. 

[15] Some altemative methods of defining a correction matrix impose constraints on the form 
of the correction. For example, in some methods, the correction matrix must be diagonal rather 
than fully-populated. In other methods the gain values are defined so as to map a neutral object 
under the estimated illuminant to neutral values under the reference illuminant. For each form of 
correction matrix, it is necessary to map from estimated illumination parameters to correction 
parameters. 

[16] Self-luminous or non-reflecting image regions can lead to problems including incorrectly 
biasing the illuminant estimate or problems in color correction due to violations of the 
assumptions underlying the color correction transform. 

[17] In some embodiments of the present invention, image elements, such as pixels or groups 
thereof, are evaluated to determine whether they are likely to represent self-luminous objects in 
the image. These elements may be more likely to be self-luminous when they are located in 
proximity to the edges of the image. For example, and not by way of limitation, in images that 
contain regions of sky, a self-luminous area, the self-luminous sky area is typically located along 
the top edge of the image. When images are rotated, such as from landscape to portrait format. 
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top edges may become side edges and vice versa. Accordingly, in some embodiments of the 
present invention, proximity to any edge can be an indicator of self-luminance. In other 
embodiments, image orientation algorithms may be used to identify the top, bottom and sides of 
an image. In other embodiments, an image may comprise metadata or some other indication of 
image orientation. In these embodiments with identified orientation, proximity to particular 
edges may be indicative of self-luminance. For example and not by way of limitation, a pixel's 
proximity to the top of the image may be more indicative of self-luminance than proximity to the 
side or bottom of the image. 

[18] In some embodiments, an element's likelihood of being self-luminous may be determined 
in relation to the luminance or brightness of the element. In some embodiments, the output of 
this function can be viewed as a continuous-tone grayscale image, where lighter regions 
correspond to regions likely to be self-luminous objects and darker regions correspond to 
reflecting surfaces. 

[19] In some embodiments, an element's chromaticity or color characteristics can be used to 
determine self-luminance. When an element's chromaticity is similar to the chromaticity of 
known illuminants or common self-luminous objects, that element is more likely to be self- 
luminous. 

[20] Some embodiments of the present invention comprise the steps illustrated in the chart of 
Fig. la wherein an image element, such as a pixel or region of interest, is evaluated 10 to 
determine its characteristics such as location, chromaticity or color characteristics and/or 
luminance value. The location of the element is evaluated 12 to determine its proximity to image 
edges and, if the element is sufficiently proximate to an edge, a specific edge or some other 
geometric image boundary the element is considered more likely to be self-luminous and a flag 
or decision variable value is set 1 1 or incremented according to this likelihood. 
[21] The color characteristics of the element are then evaluated 14 and the element is 
considered more likely to be self-luminous if the color characteristics match those of known 
illuminants or self-luminous objects. Color characteristics comprise chromaticity, luminance and 
other attributes. When these characteristics indicate a likelihood of being self-luminous, a flag or 
decision variable value is set 13 according to this likelihood. 
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[22] These evaluations 12, 14 and the associated flag or decision variable setting may be 
performed in any order as needed for specific applications. The results of these evaluations can 
be combined 16, such as by variable addition. Boolean logic or by other methods, and, if 
sufficient evidence exists, the element can be considered likely to be self-luminous 18. 
[23] In other embodiments of the present invention, as illustrated in Fig. lb, an image element 
can be evaluated and classified according to a weighting factor or score. In these embodiments, 
an element is evaluated to determine whether its location is proximate to image edges, specific 
edges or some other image boundary characteristic. If the location is sufficiently proximate to 
one of more of these characteristics, the weight factor or score of that element is increased 24 and 
the element is further evaluated, 

[24] If an element's color characteristics are similar to those of known self-luminous objects 
26, such as blue sky, overcast sky, tungsten-incandescent light, fluorescent light and others, that 
element can be considered likely to be self-luminous and the element's weighting factor or score 
can be increased 28 to reflect this likelihood. 

[25] If an element's luminance characteristics are found to be similar to those of known self- 
luminous objects 30, the weighting factor or score of that element can be increased 32 to reflect a 
higher likelihood that the element is self-luminous. When the score or weighting factor of an 
element is sufficiently high, the element may be considered self-luminous. 
[26] The methods of some embodiments of the present invention may be illustrated with 
reference to Fig. 2, which is an original digital image and Figs. 3a-3d, which is a set of images 
derived firom Fig. 2 depicting the likelihood of self-luminosity based on element location, 
chromaticity or color characteristics, and/or luminance parameters. In the images in Figures 3a- 
3d, white areas represent areas of high likelihood, while black areas represent low likelihood. 
[27] In some embodiments of the present invention, the self-luminous decision variable, s, can 
be computed from the pixel color values, x, and pixel position, (i j), as: 
s = f(x,i,j) 

[28] In some embodiments, independence between the factors can also be assumed. This is 
convenient computationally at the expense of not taking advantage of correlations between terms 
(e.g., that sky is likely to be both bluish and of high luminance). 
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s=g(x)*h(io) 

The term dependent on color values can be decomposed into separable terms based on 
chromaticity, c, and luminance, Y: 

s = gi(c)*g2(Y)*h(ij) 

[29] For gi(c), a function can be designed that has a maximum at the hue angle of "blue sky" 
and that decreases as the hue angle deviates from this value. We base g2(Y) on luminance 
distributions for a large number of sky regions. For h(ij), the simple assumption can be made 
that sky is more likely near the edges of the image. Here h(i,j) is a linear function of pixel row 
number. Note that these choices for gi(C) and g2(Y) strongly bias the model toward identifying 
regions of "blue sky" or another illuminant or group of illuminants as self-luminous objects. 
[30] Fig. 2 is an image containing surface regions 42 and sky regions 40. Fig. 3 illustrates a 
decision variable map for each self-luminosity likelihood function. We use "xy" to denote 
chromaticity and "Y" to denote luminance, after the CIE 1931 xyY colorspace. In the ground 
regions, the final decision variable is near zero as illustrated by the black color in all four images 
in Figs 3a-3d. 

[31] In Fig. 3a, the likelihood of an element being self-luminous near the top edge of the 
image is shown by the lighter shading that gets progressively darker as the location becomes 
more distant from the top edge. 

[32] In Fig. 3b, the elements or pixels that have chrominance similar to "blue sky" or another 
illuminant or group of illuminants are shown in white, while those "distant" from that range of 
hues are shown in darker shades. 

[33] In Fig. 3c, the elements or pixels that have luminance values similar to those of known 
self-luminous objects are shown in lighter shades. In this exemplary image case, few pixels have 
high luminance as the sky in this image is relatively dark. 

[34] Fig. 3d shows a composite image representing a combination of images shown in Figs. 
3a-3c. In this figure, only elements in the sky are shown in lighter shades thereby signifying a 
successful application of the combined analysis. 
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[35] Fig. 4 is a diagram illustrating the distribution of self-luminosity decision variables 
corresponding to Fig. 3d. The "sky" region corresponds to the values in the 0.2 - 0.35 range. 
[36] Embodiments of the present invention may be used in conjunction with color balance or 
color constancy estimation methods and color balance correction in digital images. In some 
embodiments of the present invention, as shown in Fig. 5, inventive methods and systems may be 
used to determine self-luminous elements 110. These methods and systems may further 
influence the determination of an image illuminant 115 and the correction used to achieve color 
balance 120. These systems and methods may also have an effect on the application of the color 
balance correction 130. 

[37] Most color-balance algorithms can be modified to use a self-luminous classification map 
to derive color-balance parameter estimates from only those pixels classified as non-self- 
luminous. These modified algorithms would produce estimates with reduced bias due to the 
presence of self-luminous image regions. 

[38] For example, a greyworld algorithm can be modified to compute a weighted average of 
the scene chromaticities where the weight would be reduced for pixels likely to be self-luminous. 
[39] Consider the illuminant estimation method that selects the illuminant parameters Cj such 
that the function f(ei) is maximized for the image, I: 
f(ei) = Zp(xklI,Sk=0)*p(xJei) 

where 

Xk represents a vector of color values 

Sk represents a binary variable indicating the state of self-luminous (S=l) or surface (S=0) 

Ci represents a set of parameters for the ith illuminant 
p(xk 1 1, Sk=0) represents the likelihood of observing color coordinates Xk in an image, I, for the 
non-self-luminous pixels. p(xk I ei) represents the likelihood of observing color coordinates Xk 
under illuminant ei. This term represents a scene/image color probability model that is indexed 
by illuminant. When the color frequencies observed in the image match the color frequencies 
expected under a particular illuminant, the value f will be large. The illuminant corresponding to 
the maximum of f(ei) is the illuminant most consistent with the surface pixels in the image. 
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For a discrete set of color coordinates, the observed color likelihood function can be 
approximated as: 

p(xkll, S=0) = nk/Ns=o 

where 

nk represents the number of surface pixels within a distance e of coordinate Xk 
Ns=o is the number of pixels in the surface class 

[40] An alternative method of estimating an illuminant from an image, I, is to directly use the 
self-luminous decision variable rather than a classification map. Consider the illuminant 
estimation method that selects the illuminant parameters ei such that the function g(ei) is 
maximized: 

g(ei) = 2:wk*p(Xklei) 

where 

Wk represents a weight for a kth vector of color values that is dependent on the 
corresponding self-luminous decision variable 
[41] The value Wk may be computed from pixel self-luminous decision variable, s: 
Wk = 1-s 

where s is normalized to have range 0-1. 

[42] p(Xk I ei) represents the likelihood of observing color coordinates Xk under illuminant ej. 
The term Wk weights the contributions of pixels that are non-self-luminous more than pixels that 
are self-luminous. Thus the value g(ei) will be a measure of how consistent illuminant Ci is with 
the surface pixels in the image. 

[43] For a discrete set of color coordinates, the weight may be computed as: 
Wk = Z Wj 

where 

j indexes over the pixels within a distance e of coordinate Xk 

[44] In some embodiments, Bayesian color balance estimation techniques formulate the 
problem as selecting die illuminant, ei, that maximizes the illuminant posterior probability 
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distribution, p(ei 1 1), of the illuminant parameters conditioned on the image data. Bayes' Law 
relates the posterior to the prior, p(ei), and image likelihood function, p(I I ei): 

p(e,|l) = *p(l|e,)p(^,) 

where 

k is a normalizing constant 

p{ei ) represents the likelihood of different illuminants 
p(I I ei) represents the likelihood the image was illuminated by illuminant . 
[45] If pixels are assumed independent from one another, then the color likelihood of the 
image is the product of the likelihoods of each pixel's color values, xj, given the illuminant: 

p(Ilei) = np(xjlei) 

[46] One method of introducing a self-luminous classification map into the pixel likelihood 
function is to let a binary variable S indicate that a pixel is self-luminous (S=l) or non-self- 
luminous (S=0). The color likelihood function is equal to, 

p(xj I eO = p(xj I Ci, S=0)p(S=0) + p(Xj I ei, S=l)p(S=l) 
[47] For the case where a pixel is self-luminous, the color likelihood function can be treated as 
conditionally independent of the illuminant: 

p(xjlei,S=l) =p(XjlS=l) 

where 

p(xj I S=l) is a probabilistic model for self-luminous regions 
Substituting, the image likelihood is computable as: 

pa I Ci) = n {p(xj I ei, S=0)p(S=0) + p(xj I S=l)p(S=l)} 
From the image likelihood function, the illuminant posterior distribution may be computed and 
then maximized to yield an illuminant estimate. 

[48] In some embodiments, the posterior, p(ei 1 1), may be computed by multiplying the image 
likelihood and illuminant prior. The prior, p(ei), may be used to account for the relative 
likelihood of different illuminants; for instance, illuminants may be expected to be near the 
daylight locus. 
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[49] In some embodiments, a cost function may be used and computed by multiplying the 
posterior by a function, which biases the result toward a reference illuminant (such as CIE 
D6500). A bias toward a reference illuminant can be useful because corrections associated with 
extreme estimated illuminants can produce undesirable resultant images if the illuminant 
estimate is in error. 

[50] The illuminant estimate may be the illuminant associated witii the maximum after 
applying the cost function. 

[51] The color likelihood function given the illuminant parameters, p(xj I ei), is computed by 
making assumptions or measurements of the distributions of colors in scenes. The purpose of 
this term is to characterize the likelihood of different colors under each illuminant. For example, 
under a reddish illuminant chromaticities with a nominally reddish hue would be more likely and 
nominally bluish chromaticities would be highly unlikely. 

[52] Fig, 6 illustrates how a set of matte Munsell surfaces shifts under a set of illuminants that 
are near the daylight locus. Each colunm of plots corresponds to a different illuminant. 
[53] Fig. 7 illustrates the chromaticities of the illuminants depicted in Fig. 6. The color of 
these illuminants varies in a bluish-yellowish direction and there is a clear shift in the 
chromaticities of the Munsell set with each illuminant. One method of computing p(xj I Ci) is to 
assume that colors in images are uniformly drawn (witii replacement) from the set of Munsell 
matte patches. The long-term average color frequencies will be the same as rendering the 
Munsell set under the illuminant. The probability of a given chromaticity, [x,y], under the ith 
illuminant will simply be the frequency at that chromaticity of the distribution of the Munsell set 
under that illuminant. An alternative method is to derive an emprical color model from the 
surface color distributions across a large collection of images or objects. 

[54] In some embodiments of the present invention, a likelihood of occurrence of specific 
colors may be pre-computed for a set of illuminants. These may be referred to as model gamuts, 
p(chromaticity I ei). The model gamuts represent the probability of occurrence of each 
chromaticity given the illuminant. 

[55] The probability of being self-luminous is computed for each pixel, p(S=l I chromaticity, 
luminance, position). This probability is based on the pixel's position in the image, its 



-14- 



luminance, and its chromaticity or color characteristics. The inverse values, p(S=0 I 
chromaticity, luminance, position), are accumulated as a function of chromaticity. The result of 
this accumulation is a histogram of p(S=0) as a function of chromaticity, h(chromaticity). 
[56] In some embodiments, the image likelihood, p(I I ei), may be approximated as the inner 
product between h(chromaticity) and p(chromaticity I ei). The image likelihood is computed for 
each illuminant. Across all illuminants, this is a function of illuminant chromaticity, 
L(chromaticity). 

[57] Some embodiments of the present invention may be explained with reference to Fig. 8, 
which is a flowchart illustrating the steps of an image illuminant estimation process. Although 
the method is depicted as a sequence of numbered steps for clarity, no order should be inferred 
from the numbering unless explicitly stated. It should be understood that some of these steps 
may be skipped, performed in parallel, or performed without the requirement of maintaining a 
strict order of sequence. Initially, image element characteristics are obtained 200. These 
characteristics typically comprise image element color characteristics, luminance characteristics 
and location data as well as other characteristics. Using the element characteristic data, the 
likelihood that each image element is self-luminous can be determined 202. This may be 
performed as explained in other embodiments above with reference to image element proximity 
to image edges or boundaries, to element chromaticity relative to chromaticity of known 
illuminants and to element luminance. 

[58] A weightmg factor may be assigned 204 to each element based on its likelihood of being 
self-luminous. The most likely illuminant is then estimated 206 for each element based on 
element characteristics. The image illuminant is then estimated 208 using image element 
characteristics and the weighting factors. The image illuminant may be estimated 208 in many 
ways according to the methods of embodiments of the present invention. In some embodiments, 
the weighting factor may be used to adjust the contribution of elements to the image illuminant 
estimation process. This adjustment may comprise complete omission of likely self-luminous 
elements or diminution of their contribution to the estimation process. 

[59] Other embodiments of die present invention may be explained with reference to Fig. 9, 
which is a flow chart showing an image color-balance correction process. In these embodiments. 
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iraage element characteristics are obtained 220 in a typical manner as is known in the art. A 
likelihood that each image element is self-luminous is then determined 222 according to methods 
explained herein. A global image illuminant is estimated 224 for the entire image based on 
image element characteristics and the likelihood that each element is self-luminous. A correction 
factor is then computed 226 to color-balance the image for the illuminant that has been 
estimated. This correction factor is then applied to the image 228 to achieve proper color- 
balance. 

[60] Other embodiments of the present invention may be explained with reference to Fig. 10, 
which is a flow chart illustrating a method for selective application of a color-balance correction 
factor. In these embodiments, image element characteristics are obtained 240 by methods known 
in the art. A likelihood that each element is self-luminous is determined 242 as explained above 
and an image illuminant is estimated 244 based on the element characteristics and the likelihood 
that each element is self-luminous. One or more correction factors are computed 246 to correct 
the image for the illuminant and the correction factor or factors are applied to the image 248 
according to each element's likelihood of being self-luminous. In these embodiments, the 
correction factor may be applied only to elements that are not likely to be self-luminous, may be 
applied in degrees according to magnitude of the likelihood of being self-luminous or may be 
applied in other ways that are proportional or otherwise related to the likelihood of an element 
being self-luminous. 

[61] Systems and methods for determining a likelihood of an image element being self- 
luminous, computing correction parameters and applying digital image correction parameters 
have been explained herein. A few examples have been given to illustrate and clarify aspects of 
the invention. However, the invention is not Umited to merely these examples. Specific 
correction algorithms have also been given to provide a context for the invention, but again, the 
invention is not limited to just the mentioned algoritiims. Other variations and embodiments of 
the invention will occur to those skilled in the art. 



[62] What is claimed is: 



